Online image retention, indexing, search technology with integrated image licensing marketplace and a digital rights management platform

ABSTRACT

An online image search and marketplace platform and browser plug-in system is described. The system is configured to execute image-based searches of the Internet to correlate hosted images with ownership and licensing metadata, facilitating the marketing of images to users and expediting the purchase of licensing for said images. The system employs an online platform in communication with an image metadata database, providing a conduit for users to execute an image search from an image search field populated by a target image URL or direct image upload. An auto-cropping feature combined with extensive steganography integration ensures accurate image identification, ownership mapping, and usage tracking. The browser plug-in facilitates the execution of an image search within the right-click menu of any image hosted to the Internet. Image tracking enables content owners to be alerted in real-time when unlicensed use of content is detected.

CONTINUITY

This application is a non-provisional application of provisional patentapplication No. 62/173,440, filed on Jun. 10, 2015, and priority isclaimed thereto.

FIELD OF THE PRESENT INVENTION

The field of the present invention relates to the retention, finding,organization and distribution of proprietary image data or otherwiseprotected images, and more specifically relates to a process ofsearching for images online regarding licensing, use, and ownershipmetadata, coupled with an online interface configured to expedite thepurchase of a license for use of said images by a user, and to notifycontent owners of unlicensed use of content online.

BACKGROUND OF THE PRESENT INVENTION

It is known that the Internet continues to grow in size and applicationexponentially every day. The sheer number of newly uploaded pieces ofcontent to the Internet, from news and blog articles, to professionallydesigned graphics and personal pictures, is being expanded every minute.Many websites hosting news, blogs, and similar web articles frequentlyaccompany stories and posts with images to further entice the reader andto contextually and visually engage him or her more deeply into thestory. Commonly, writers and editors employ the Internet to search forimages that pertain to the story, which they are looking to publish. Perconvention and US law, images that are in the public domain maygenerally be used without permission; and may even be used for profit inmany cases. Images licensed under the Creative Commons license may belicensed for use, often for free, with permission, but requireattribution to the owner of the image license. Oftentimes, these imagesmay not be used for profit. Other, non-copywritten and non-registeredimages are frequently uploaded online without formal protection.However, in all cases, permission to use such images should beascertained prior to public use.

Conversely, owners of images online commonly experience difficultydetermining when and if their images are in use without permission,which would be an act of infringement of the intellectual property ofthe owner. Unless a content owner personally scours the Internet todetect infringement, it remains unlikely that the creator and/or ownerof the image would be made aware of infringement in a timely manner. Inmany cases, infringement remains unchecked, as it is the duty of theowner to call out and prosecute, if necessary, cases of non-licensedinfringement of his or her intellectual property. If there were a meansby which image owners would be automatically notified of unlicensed useof their content, cases of infringement would decline, and prosecutionof infringement cases may be served more expeditiously.

Given the staggering size of the ever-growing Internet, it has becomeincreasingly difficult for writers, editors and publishers to identifythe true owners of many images online. Therefore, requesting or payingfor permission to license these images can be nearly impossible unlessthe image is hosted and/or supplied by a registered image marketplacesuch as www.istockphoto.com, www.bigstockphoto.com, or other stock photoor similar image marketplaces. If a user cannot determine ownership ofan image online, payment for a license is impossible, leading many usersto unlawfully use the image without permission. Ideally, there shouldexist a convenient, seemingly omniscient image marketplace and querysystem capable of returning image ownership, licensing, and othermetadata at will in real-time to facilitate an image license purchase.

A small number of image search services are available on the markettoday that provide a means for users to ascertain the origin metadata ofan image online via the image URL, or by uploading a copy of the image.Unfortunately, nearly all of the image search services available todayare inaccurate, because they are solely using key words and meta data toascertain a match for the image or those that do not are just not veryaccurate. The two competitors that have the best current availabletechnology that are not solely reliant on key words and meta are Google™and TinEye™. Google of course has the advantage in producing betterimage search results, as it has billions of images, and therefore hasextensive training data. Training data is the use of previously definedimages to train a computer system to understand what to look for whenlooking for a particular object or part of an object (the renowned blindman describing the elephants trunk).

However, Google does not provide sub-filtering of image search results,and also does not allow a user to frame a portion of the image of whichthey are most interested in, and Google continues to primarily use amethod of image identification in association with a conventional bag ofvisual words search technology (histogram representation based onindependent features of an image) while ignoring most available methodsfor image similarity identification, nor does Google provide an API toallow third party access to the Google's image search technology. Tineyedoes have an API, and does license its image search technology. However,Tineye does not have nearly as large a database as Google, and thus hasa very small amount of training data (estimated only about 10 millionimages), and only uses known algorithms vs customized ones. Therefore,Tineye can provide no better image search results and ranking thanGoogle, as Tineye does not have the same amount of images stored on theGoogle datacenter (estimated to be more than 1 trillion), Tineye cannotprovide any meaningful nor comprehensive correlation of image ownershipto images across the Internet. A new means of storing images, or ratherthe essence of an image, is required in order to minimize databasestorage required for competent image search execution, and to expeditethe image search process.

Without easy location of the image owner, licensing for use iseffectively impossible. Conversely, without a means for image owners todetect when their property is in use online, image infringement commonlygoes unnoticed, and therefore unprosecuted and certainly un-monetized.The exponentially growing number of images online further exacerbatesthis problem.

Although the science of steganography is known, it is not used by anymass online image searching tool nor image matching service. If the useof steganography was employed in a component of an image marketplace,image identification, and subsequent licensing/sale would befacilitated.

Thus, there is a need for a new, more accurate image search and imagemarketplace system configured to facilitate the image storage, theidentification of images correlated to ownership metadata, the licensingof images for use, and the alerting of content owners of existing imageinfringement in real-time. Such an image marketplace system wouldpreferably be configured to facilitate the communication and potentialtransaction between image buyers/licensors and image owners.

SUMMARY OF THE PRESENT INVENTION

The present invention is a system and method for the storage,identification, categorization, filtering, licensing, and distributionof images online via an image search portal and image licensing browserplug-in. An online platform is employed to facilitate the searching ofimages by a user based on a target image accessible via the image URL oran uploaded image copy. The online platform is configured to execute asearch of the Internet for (a) the image itself, (b) similar images andto correlate information about the image with all available datapertaining to the use of the target image being searched for, as well asimage metadata relating to the ownership, licensing status, distributionoptions, and purchase information (available to the non-owner of thetarget image, while also providing income information to the owner ofthe target image. The target image may be automatically cropped ormanually cropped within the platform of the present invention to moreeasily search for uses of a specific portion of the image online,facilitating more effective image mapping.

The storage of large numbers of images is a problem, as it createsmassive and ongoing storage costs. The platform of the present inventionhas been designed such that it does not require the storage of theactual digital images. The solution provided by the present invention isto take an image in its original form, index its unique features, andstore them as vectors inside the index without duplication acrossfeatures from different images. These features are then removed,creating a shell of the image excluding the elements that make the imagestorage large. The shell is then stored. If the original image isrequired for display, the indexed features are applied to the shell,reproducing the original image with all of its elements.

The system of the present invention additionally employs a browserplug-in, which integrates a quick-access menu into the right-clickdrop-down menu of a hosted image displayed within the Internet browser.The browser plug-in enables a user to right-click on any image on theInternet and quickly ascertain information relating to the use of theimage, and provides a direct avenue to purchase a license for use of theselected (clicked) image and/or save the image as an image of interestin the user's account in the platform of the present invention. Ineffect, this application of the system of the present inventionostensibly converts the Internet itself into an integrated imagemarketplace, as every image displayed to a user within the browserwindow equipped with the browser plug-in of the present invention may beright-clicked on a computer (or tapped and held on a tablet or mobiledevice), presenting the user with image information and licensingoptions. Users are also presented with a direct avenue to clear therights to images quickly and easily if applicable. Internal to theapplication itself, the user may drag numerous images to the searchfield and see the ownership information at a glance, and even buy allrequired rights for the users' application as required in bulk. As videois merely a large number of images, this can be done with video as well.

In short, the solution performed and executed by the system of thepresent invention is as follows: The system employs a new set ofalgorithms and processes which facilitate the storing and identificationof images by using (a) a new image storage method, (b) image filteringbased on an array of algorithms, image elements, and filters rather thanthe conventional “bag of visual words” model, (c) a free image stampingsystem which cross indexes images with ownership information, and (d) acomprehensive browser plug-in to make the latter information easilyaccessible by a user.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be better understood with reference to theappended drawing sheets, wherein:

FIG. 1 exhibits a flow chart detailing the back end processes executedby the system of the present invention during use.

FIG. 2 is a flow chart that details the method of use of the presentinvention by a user searching for images from the platform of the systemof the present invention.

FIG. 3 displays an example screenshot of the image search andidentification online platform.

FIG. 4 exhibits a flowchart detailing the process of use of the presentinvention by a user seeking to inquire about, and purchase a licensefor, an image via the browser-plug in of the present invention.

FIG. 5 shows the integration of the plugin of the system of the presentinvention into the right-click menu within a browser.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

The present invention generally comprises a system and method configuredto expeditiously identify an image online via an executed image search,return image metadata, including but not limited to ownershipinformation, licensing status (fair use, Creative Commons, etc.),creation date, instances of licensed and unlicensed use, and EXIF data(if applicable), and facilitate purchasing of a license to use the imagedirectly via a web interface, mobile device or integrated browserplug-in (20). The system of the present invention is embodied either byan online platform (10) and the integrated browser plug-in (20), orthrough an API, which provide a means for users or systems to access theimage search system (back end which preferably uses an Apache Solr andLucene platform) of the present invention. An image metadata database ispreferably employed by the present invention to aggregate searched imagedata, facilitate machine learning processes which expand the systemsreach through clustering, auto-tagging, recommending and expeditingfuture access, helping to ensure a consistent record of use for eachimage. The metadata database may have any information about the imagerequired including, but not limited to ownership information, ownershiptransfer/assignment information, copyright numbers, image specificinformation such as colors, size, date, place, resolution, etc. Theimage metadata database is pinged and updated with each image searchexecuted via the system of the present invention.

The online platform (10) of the present invention is preferably equippedwith an image search field (30) configured to be populated with an imageURL or file manager address of an image for upload. Once an image hasbeen targeted via upload or image URL in the image search field (30),the system of the present invention executes an image search for any andall instances of the image on the Internet, as well as closest matches,then returns and displays the results to the user. It should beunderstood that the present invention facilitates the purchase, sale, orlicensing of a digital image via the online platform (10) as well as viathe integrated browser plug-in (20), preferably via a ‘license image’button, ‘buy image’ button, ‘file infringement claim’ button, and otherpertinent buttons relating to image marketplace functionality.

Upon display of the results, the user may opt to crop the image andexecute a secondary search on the modified image, filtering the resultsby the closest or most relevant matches. Filtering will also beavailable with a focus on different feature set elements (color, hue,line, etc.) not based on cropping. Additionally, the system of thepresent invention employs an auto-cropping feature configured toautomatically crop an image according to an assumed focal point of theimage determined via an internal image reading algorithm. Feature setsmay also be chosen manually via the interface to help algorithm focus.Any steganography present in the image is employed to best match theimage to copies in use or hosted online as well as use of theinformation in the meta database. EXIF data is employed to match andassociate image copies if present. Users of the system of the presentinvention via the online platform (10) may arrange to purchase a licenseto use an image or images once the owner of the image is detected by theimage search algorithms or steganography information in the image of thepresent invention.

Algorithms are used as the best methods for a user to perform thecurrent search based upon user requirements. Search algorithms employedby the present invention preferably include: CEDD, ScalableColor,ColorLayout, PHOG, Edges, Opponents, JCD, FCTH, FuzzyOpponentHistogram,JointHistogram, Surf, Sift, Gabor, Tamura, LuminanceLayout,JpegCoefficientHistogram, SimpleColor, ILocalBinaryPatterns,RotationInvarientLocalBinaryPatterns, and/or BinaryPatternsPyramid.Additionally, using machine learning processes, the system classifiesthe images based on its stored vector data as well as any EXIF data,keyword, and tagging information present, which compliments the searchalgorithms by allowing comprehensive searching based inside classified(categorized) groups.

Use of the system of the present invention by a user is preferablyconducted via the online platform (10) and/or the integrated browserplug-in (20) of the present invention. Under standard circumstances, useof the present invention is preferably enacted as follows:

First, a user selects an image, either online or from his or her harddrive. (300) The user then enters the location information (image URL orlocal file manager address) of the image into the image search field.(310) If the image is selected from the user's local computer or mobiledevice, the image is uploaded to the system. (320) The image may becropped for a target image area automatically or manually by the userprior to search execution. (325) The user executes the image search,causing the system of the present invention to conduct an extensivesearch of the Internet for any and all instantiations of the searchedimage. (330) The system returns a list of results of all instances ofuse of the searched image, complete with a screenshot of the image inuse online. The results include image metadata pertaining to ownership,licensing status, EXIF data, and history. (340) The image may beadditionally cropped (or for the first time) preceding a secondary imagesearch if required. (345) The system presents the user with the optionto purchase a license to use the searched image after ownership of theimage is established and linked to the image via a direct URL, or withinthe platform of the present invention itself (350) Steganography isintegrated into the uploaded image to expedite accurate searches of theimage in the future. (360) In the event that the user is the owner ofthe image, the user is notified of instances of unlicensed use of his orher intellectual property, alerting him or her to the infringement sothat action make be taken in the form of a Cease action or eventualprosecution. (370)

In a second scenario as seen in FIG. 4, the browser plug-in component ofthe system of the present invention is preferably employed by a user asfollows:

While browsing and/or searching the Internet, a user comes across animage ideal for use in a marketing proposal or blog article, and seeksinformation pertaining to licensing the image for use. (400) The userright-clicks on the image within the Internet browser and selects “ImageProperties,” “purchase image license,” “Image Inquiry,” or “Look upImage” within the right-click drop-down menu. (410) An image search forthe image is executed by the system of the present invention. (420)Image information history, licensing information, ownership information,and other image metadata are returned and displayed to the user. (430)An option to purchase a license to use the image is presented to theuser if the image is available for licensing. (440) The user purchasesthe license within the online platform of the present invention. (450)Payment is transferred to the owner of the image. (460) A small fee maybe assessed for the service provided by the present invention which isto be integrated into the licensing fee. (470)

It should be understood that the system of the present invention iscapable of executing searches for images collected in batches orcompressed files, performing a search on each individual image after thebatch is uploaded, and displaying the results in order by date (or otherfilter) to the user.

The back-end processes performed by the system of the present invention,as shown in FIG. 1, are preferably executed roughly as follows:

First, the system receives an image or the URL of an image. (100) Thesystem then processes a search across all algorithms (as listed above).(110) The results are collated by score and distance from the sourceimage. (120) The results are then returned to the user. (130) If thesource image requires cropping, the image is cropped using a circle,square, or rectangular cropping tool, and the search is executed again.(140) If no cropping of the source image is required, the system queriesif the results require filtering. (150) If filtering is necessary, theresults are filtered based on all filtering mechanisms as listed above.(160) The search is executed once more pending the completion offiltering. When filtering and the subsequent search is complete and nolonger needed, licensing is queried. (170) If the results dictate thatthe image is available for licensing purchase, the license may bepurchased, wherein license information is embedded into the image usingsteganography. (180) The image is then delivered to the user via HTTPdownload, FTP download, Cloud Service upload, or email. (190) If theuser opts to not purchase the license, the process service is ceased.(195)

In terms of image compression for search, correlation of results of thesearch, and reconstruction of the source image, a first embodiment ofthe back-end processes of the present invention includes:

First, the system ingests the source image. Then, the source image isconverted to grayscale. The source image is then split into overlappingpieces. Each of the overlapping pieces are then resized such that eachpiece is no larger than 32×32 pixels, and no smaller than 8×8 pixels.The image pieces are indexed, and each piece is trained being the sourcevia the use of interest points, histograms, and hashes of internal imagedata, correlating to points for reconstruction of the source image.Additionally, the system of the present invention may detect andrecognize objects in the source image prior to indexing, classifying,and producing a master set of groups and categories. It should beunderstood that source images are preferably compressed down to basiccolors on a 256 color scale for search and classification by color data.

An additional alternate embodiment of the back-end process of thepresent invention includes:

First, the source image is ingested by the system. Then, the systemremoves the background of the source image by polarizing or creating itsopposite, building a foreground mask of the source image to act as atemplate. Then, the foreground mask or template is applied, whereby thebackground of the image is removed, leaving only the foreground objects.Then, the foreground object lines are detected, creating new images ofeach object. Histograms are then created and interest points aredetected, which are indexed and classified for image search parameters.

Having illustrated the present invention, it should be understood thatvarious adjustments and versions might be implemented without venturingaway from the essence of the present invention. Further, it should beunderstood that the present invention is not solely limited to theinvention as described in the embodiments above, but further comprisesany and all embodiments within the scope of this application.

The foregoing descriptions of specific embodiments of the presentinvention have been presented for purposes of illustration anddescription. They are not intended to be exhaustive or to limit thepresent invention to the precise forms disclosed, and obviously manymodifications and variations are possible in light of the aboveteaching. The exemplary embodiment was chosen and described in order tobest explain the principles of the present invention and its practicalapplication, to thereby enable others skilled in the art to best utilizethe present invention and various embodiments with various modificationsas are suited to the particular use contemplated.

We claim:
 1. A method for facilitating identification of ownership andlicensing of digital images and subsequent commerce of the digitalimages between image owners and image users on a browser-based imagemarketplace system, wherein the system comprises a server computer and adatabase storing metadata of the digital images, the method comprising:receiving, by the server computer, a selected digital image from animage user, wherein the received digital image is a source imagecomprising a source image file, wherein the source image is an image ora URL of the image; processing, by the server computer, the sourceimage, wherein the processing the source image comprises the followingsteps: converting the source image to grayscale; splitting the convertedsource image into overlapping pieces; and resizing the overlappingpieces, wherein the resizing the overlapping pieces further comprisesresizing each of the overlapping pieces to be no larger than 32×32pixels and no smaller than 8×8 pixels; indexing, by the server computer,the each piece of the overlapping pieces with interest points,histograms, and hashes of internal image data, wherein the indexing theoverlapping pieces further comprises correlating the indexed interestpoints, the indexed histogram and the indexed hashes to points of thesource image; generating, by the server computer, parameters of thesource image for an image search by aggregating metadata of the sourceimage from the source image file, wherein the aggregated metadataincludes image color, image hue, image size, image date, image location,image resolution, and image capture device; executing, by the servercomputer, the image search for instances of the source image over theInternet based on the generated parameters of the source image;returning, by the server computer, results of the image search of thesource image, wherein the results comprise image metadata of the imagesearch relating to ownership and licensing status; displaying, by theserver computer, the results of the image search to the image user;based on the results of the image search, providing, by the servercomputer, licensing information of the source image to the image userfor a license purchase of the source image; receiving, by the servercomputer, from the image user, information of the license purchase ofthe source image; creating, by the server computer, a shell of thesource image using compression and-the-indexed histograms by detectingthe interest points of the source image; storing, by the servercomputer, the shell of the source image in the database; and updating,by, the database with the image metadata of the image search of thesource image.
 2. The method of claim 1, the method further comprises:integrating, by the server computer, steganography in the source image,wherein the integrating the steganography further comprises embeddingthe license information into the source image.
 3. The method of claim 1,wherein the source image is an image uploaded by the image user.